Measuring what matters in healthcare: a practical guide to psychometric principles and instrument development.

The provision of quality healthcare relies on scales and measures with robust evidence of their psychometric properties. Using measurement instruments with poor reliability, validity, or feasibility, or those that are not appropriate for the target diagnostic group or construct/dimension under consideration, may be unfavorable for patients, unproductive, and hinder empirical advancement. Resources from the COnsensus-based Standards for the selection of health status Measurement INstruments (COSMIN) group can assist in identifying and developing psychometrically sound measures. The COSMIN initiative is the only international, research-based practice taxonomy and methodological guidelines for measurement in healthcare. This manuscript aims to provide an accessible introduction to theories, principles and practices of psychometrics, instrument properties, and scale development, with applied examples from the COSMIN recommendations. It describes why measurement in healthcare is critical to good practice, explains the concepts of the latent variable and hypothetical construct and their importance in healthcare assessments, explores issues of flawed measurement and briefly explains key theories relevant to psychometrics. The paper also outlines a ten-step process to develop and validate a new measurement instrument, with examples drawn from a recently developed visuoperceptual measure for analysis of disordered swallowing to demonstrate key concepts and provides a guide for understanding properties of and terminology related to measurement instruments. This manuscript serves as a resource for healthcare clinicians, educators, and researchers who seek to develop and validate new measurement instruments or improve the properties of existing ones. It highlights the importance of using psychometrically sound measurement instruments to ensure high-quality healthcare assessments.

Development of a Visuoperceptual Measure for Fiberoptic Endoscopic Evaluation of Swallowing (V-FEES) in Adults with Oropharyngeal Dysphagia: An International Delphi Study.

Visuoperceptual evaluation of fiberoptic endoscopic evaluation of swallowing (FEES) is a commonly used assessment in dysphagia or swallowing disorders. Currently, no international consensus exists regarding which visuoperceptual measures to use for the analysis of FEES recordings. Moreover, existing visuoperceptual FEES measures are limited by poor and incomplete psychometric data, identifying an urgent need for developing a visuoperceptual measure to interpret FEES recordings. Following the COSMIN group's (COnsensus-based Standards for the selection of health Measurement INstruments) psychometric taxonomy and guidelines, this study aimed to establish the content validity of a new visuoperceptual FEES (V-FEES) measure in adults with oropharyngeal dysphagia. Using the Delphi technique, international consensus was achieved among dysphagia experts across 21 countries, resulting in a new prototype measure for V-FEES, comprising 30 items, 8 function testing items (i.e., specific tasks performed by patients while observing and rating items), and 36 unique operationalisations (i.e., defining items into measurable factors that could be measured empirically using visuoperceptual observation). This study supports good content validity for V-FEES, including participants' feedback on the relevance, comprehensiveness, and comprehensibility of the included items. Future studies will continue the instrument development process and determine the remaining psychometric properties using both the classic test theory (CTT) and item response theory (IRT) models.

A Visuoperceptual Measure for Videofluoroscopic Swallow Studies (VMV): A Pilot Study of Validity and Reliability in Adults with Dysphagia.

The visuoperceptual measure for videofluoroscopic swallow studies (VMV) is a new measure for analysing the recordings from videofluoroscopic swallow studies (VFSS). This study evaluated the reliability and validity of the pilot version of the VMV using classical test theory (CTT) analysis, informed by the consensus-based standards for the selection of health measurement instruments (COSMIN) guidelines. Forty participants, diagnosed with oropharyngeal dysphagia by fibreoptic endoscopic evaluation of swallowing, were recruited. The VFSS and administration of bolus textures and volumes were conducted according to a standardised protocol. Recordings of the VFSS were rated by three blinded raters: a speech-language pathologist, a radiologist and a phoniatrician. Inter- and intra-rater reliability was assessed with a weighted kappa and resulted in 0.889 and 0.944 overall, respectively. Structural validity was determined using exploratory factor analyses, which found four and five factor solutions. Internal consistency was evaluated with Cronbach's alpha coefficients, which found all but one factor scoring within an acceptable range (>0.70 and <0.95). Hypothesis testing for construct validity found the expected correlations between the severity of dysphagia and the VMV's performance, and found no impact of gender on measure performance. These results suggest that the VMV has potential as a reliable and valid measure for VFSS. Further validation with a larger sample is required, and validation using an item response theory paradigm approach is recommended.

Visuoperceptual Analysis of the Videofluoroscopic Study of Swallowing: An International Delphi Study.

INTRODUCTION: The Videofluoroscopic Swallow Study (VFSS) is a commonly used dysphagia assessment that is routinely analysed visuoperceptually. However, no consensus exists regarding which visuoperceptual measures should be used to analyse VFSSs. Current visuoperceptual measures for VFSSs are limited by poor quality and incomplete or indeterminate psychometric properties. OBJECTIVE: This study aimed to establish the content validity for a new visuoperceptual VFSS measure for oropharyngeal dysphagia in adults, by identifying relevant domains of the construct and generating items and corresponding response scales. METHODS: Consensus among experts in dysphagia and VFSS from over 20 countries was achieved across three rounds of anonymous online surveys, using the Delphi technique. Participants judged relevance and comprehensiveness of definitions of visuoperceptual domains of VFSS and the relevance of various domains to the overall construct. After reaching consensus on definitions of relevant domains, consensus on items were established using the same process. RESULTS: Participants achieved consensus on definitions of 32 domains recommended for analysis, and at least one item per domain (range 1-4). Domains selected by participants included both those which occur in existing measures and domains which have not been included in any measures to date. This study will form the basis for content validity of a new measure for VFSS. CONCLUSIONS: This first phase of developing a visuoperceptual measure of VFSS resulted in the identification of 32 domains and 60 items for oropharyngeal dysphagia. Developers can now advance to the next phase of measure construction; prototype development and psychometric testing.

Psychometric Properties of Visuoperceptual Measures of Videofluoroscopic and Fibre-Endoscopic Evaluations of Swallowing: A Systematic Review.

Fibreoptic Endoscopic Evaluation of Swallowing (FEES) and Videofluoroscopic Swallow Studies (VFSS) are instrumental assessments of dysphagia which provide videos of the internal structures of swallowing. They are commonly regarded as 'gold-standard' assessments; however, there is no consensus regarding a gold-standard measure to analyse the video recordings that they produce. Measures require sound psychometric properties to be suitable for clinical or research purposes. To date, no review of psychometric properties of FEES and VFSS measures has been undertaken or formally reported. This review assessed the quality of the psychometric properties of visuoperceptual measures of FEES and VFSS. Electronic databases were searched for studies reporting on psychometric qualities of visuoperceptual measures which are used to analyse recordings from FEES and VFSS. All dates until February 2017 were included. The Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist was used to evaluate the methodical quality of studies. The measures' overall quality was then assessed by combining COSMIN ratings with quality criteria. Forty-five studies, reporting on 39 measures, met the inclusion criteria for this review. Data about the measures' psychometric properties were very limited. Twenty-one measures had information available about reliability only, while 18 had information on up to five of the possible nine psychometric properties categorised within the COSMIN framework. The majority of the FEES and VFSS measures' psychometric properties were rated as 'indeterminate' overall, due to the small number of studies, issues with design, statistical analyses, and reporting practices of extant studies. There is insufficient evidence to recommend any individual measure included in this review as valid and reliable to interpret VFSS and FEES recordings. Further research, which utilises robust methodological design and reporting, is needed to examine the psychometric properties of measures for FEES and VFSS.

Speech-Language Pathologist Interventions for Communication in Moderate-Severe Dementia: A Systematic Review.

Purpose: The purpose of this study is to evaluate the evidence for direct and indirect interventions for communication in people with moderate-severe dementia. Method: A systematic search of the literature was conducted, as per the Preferred Reporting Items for Systematic Reviews and Meta-Analysed guidelines, across 8 electronic databases. Studies were included if they included direct or indirect interventions, which could be administered by a speech-language pathologist to people with moderate-severe dementia (defined as having Mini-Mental State Examination of ≤ 15; Folstein, Folstein, & McHugh, 1975). Studies also were required to include outcome measures, which reported on communication function or participation and/or well-being related to communication. Included studies were evaluated for methodological quality using the McMaster critical appraisal tool (Law et al., 1998). Results: Eleven studies met the inclusion criteria. Ten of these studies related to direct interventions and included cognitive stimulation approaches using group (n = 5) or individual therapy (n = 1); cognitive training, including naming therapy (n = 1) and spaced retrieval training (n = 1); and cognitive rehabilitation approaches using augmentative and alternative communication (n = 2). One study reported an indirect intervention: conversation partner training. Due to the heterogeneity of studies, a meta-analysis was unable to be conducted. A descriptive synthesis of results indicated that interventions generally resulted in positive changes to communication and related quality-of-life outcomes compared with baseline or control groups. Conclusions: Preliminary evidence was found to support communication interventions for people with moderate-severe dementia. The use of cognitive stimulation approaches, which use a group treatment model and conversation, as a therapy medium show promise as direct intervention options. Implications for clinical practice for speech-language pathologists and future research are discussed. Supplemental Material: https://doi.org/10.23641/asha.5985241.

Using Rasch Analysis to Evaluate the Reliability and Validity of the Swallowing Quality of Life Questionnaire: An Item Response Theory Approach.

The Swallowing Quality of Life questionnaire (SWAL-QOL) is widely used clinically and in research to evaluate quality of life related to swallowing difficulties. It has been described as a valid and reliable tool, but was developed and tested using classic test theory. This study describes the reliability and validity of the SWAL-QOL using item response theory (IRT; Rasch analysis). SWAL-QOL data were gathered from 507 participants at risk of oropharyngeal dysphagia (OD) across four European countries. OD was confirmed in 75.7% of participants via videofluoroscopy and/or fiberoptic endoscopic evaluation, or a clinical diagnosis based on meeting selected criteria. Patients with esophageal dysphagia were excluded. Data were analysed using Rasch analysis. Item and person reliability was good for all the items combined. However, person reliability was poor for 8 subscales and item reliability was poor for one subscale. Eight subscales exhibited poor person separation and two exhibited poor item separation. Overall item and person fit statistics were acceptable. However, at an individual item fit level results indicated unpredictable item responses for 28 items, and item redundancy for 10 items. The item-person dimensionality map confirmed these findings. Results from the overall Rasch model fit and Principal Component Analysis were suggestive of a second dimension. For all the items combined, none of the item categories were 'category', 'threshold' or 'step' disordered; however, all subscales demonstrated category disordered functioning. Findings suggest an urgent need to further investigate the underlying structure of the SWAL-QOL and its psychometric characteristics using IRT.

Health-Related Quality of Life and Oropharyngeal Dysphagia: A Systematic Review.

The negative impact of increased bolus modification on health-related quality of life (HRQoL) in persons with oropharyngeal dysphagia (OD) has been described in previous literature; however, findings may have been influenced by the severity of OD and underlying etiology. The current systematic review therefore aims to investigate the relationships between HRQoL and OD and to report on changes in HRQoL and OD following intervention. This review was conducted in accordance with the PRISMA statement. A systematic search of the literature was performed using PubMed and Embase. All available English publications up to February 2016 that reported on OD and HRQoL were included. Articles that studied OD arising from esophageal dysfunction or included less than 15 participants with OD, were excluded. Thirty-five studies met the inclusion criteria. Inverse bidirectional relationships were found between decreased HRQoL and increased OD severity. Following intervention, changes were evident through improved HRQoL with decreased OD severity. The findings of this paper highlight the importance of targeting HRQoL in patients with OD. However, because of the heterogeneity in methodology, terminology, and assessment procedures used in the retrieved articles, the generalization of study results is limited. More research investigating the relationships between HRQoL and OD is needed. Future studies should implement study designs that lead to stronger levels of evidence, quantify the severity of OD and underlying diseases, use validated measures and less ambiguous terminology.

Living with oropharyngeal dysphagia: effects of bolus modification on health-related quality of life--a systematic review.

PURPOSE: Difficulty swallowing, oropharyngeal dysphagia, is widespread among many patient populations (such as stroke and cancer groups) and aged community-dwelling individuals. It is commonly managed with bolus modification: altering food (usually cutting, mashing or puréeing) or fluids (typically thickening) to make them easier or safer to swallow. Although this treatment is ubiquitous, anecdotal evidence suggests patients dislike this management, and this may affect compliance and well-being. This review aimed to examine the impact of bolus modification on health-related quality of life. METHODS: A systematic review of the literature was conducted by speech pathologists with experience in oropharyngeal dysphagia. The literature search was completed with electronic databases, PubMed and Embase, and all available exclusion dates up to September 2012 were used. The search was limited to English-language publications which were full text and appeared in peer-reviewed journals. RESULTS: Eight studies met the inclusion criteria. Generally, bolus modification was typically associated with worse quality of life. Modifications to foods appeared to be more detrimental than modifications to fluids, but this may be due to the increased severity of dysfunction that is implied by the necessity for significant alterations to foods. The number of studies retrieved was quite small. The diverse nature of methodologies, terminologies and assessment procedures found in the studies makes the results difficult to generalise. CONCLUSION: Overall, even though the severity of dysphagia may have been a confounding factor, the impact of bolus modification on health-related quality of life in patients with oropharyngeal dysphagia appears to be negative, with increased modification of food and fluids often correlating to a decreased quality of life. Further, associated disease factors, such as decreased life expectancy, may also have affected health-related quality of life. More research is needed.